Method and system for computer guided surgery

ABSTRACT

A computer-implemented method and a system for computer guided surgery, which include a transposition of an action, planned in a virtual environment with respect to a virtual referential RP, to a physical action performed with a surgical tool in a real operating theatre environment for orthopedic surgery of a patient.

FIELD OF INVENTION

The present invention relates to computer guided surgery. This fieldcomprises a set of computer and/or physical systems designed to assistthe surgeon in planning and/or performing a surgical intervention. Itincludes the following three families of systems: navigation systems,robotic systems and augmented and/or mixed reality guidance systems.

The main elements of current computer guided surgery systems are thefollowing.

First, the creation of a digital model of the patient: either frompatient-specific preoperative data obtained by medical imaging (CT scan,MRI, X-rays, etc.) or generated during the surgery by computerprocessing that combines generic databases (e.g. anatomical atlases,statistical shape models) and the acquisition of intraoperative images(fluoroscopy, ultrasound, etc.) or data of the patient's anatomyobtained using three-dimensional localizers and palpation systems.

Second, the generation of a surgical plan from the computer processingof the digital patient model. This task is performed before theprocedure in the case where the patient model is derived frompreoperative data, or at the beginning when derived from intraoperativedata.

Third, surgical navigation based on a method of matching preoperativedata with intraoperative data. Surgical navigation allows the surgeon tovisualize the current positioning of the surgical instruments inrelation to the patient's anatomy, and to track the progress of thesurgery in relation to the surgical planning In standard navigationsystems, this information is displayed on a screen. In mixed realitynavigation systems, surgical guidance is displayed by augmented orvirtual reality through a helmet or goggles in an augmented or virtualreality manner

Fourth, a system to assist in the realization of the surgical gesture:patient-specific instrumentation (e.g. customized cutting or millingguides) can be produced from the preoperative planning Alternatively,visual decision support information is provided to the surgeon. If anelectromechanical system assists the surgeon in performing the surgicalprocedure, it is a robotic system.

This invention belongs to the field of computer guided surgery for thenavigation of robotic systems, with uni-compartmental and total kneearthroplasty as the main but not exclusive applications.

Computer-assisted robotic arthroplasty uses active or collaborativesystems (synergistic systems that combine the skills of the surgeon andthe robot to form a performance-enhancing partnership) and passivenavigation systems.

BACKGROUND OF INVENTION

Two main solutions are generally implemented in the prior art.

The first solution involves the use of markers placed on bone andsurgical instruments, combined with an optical tracking system to ensurethat real and digital reference points are matched and to know the exactposition of bones and surgical instruments in space. An example of thiskind of solution is presented in the American patent U.S. Pat. No.10,136,950 B2 which uses a surgical marker to fit the anatomy of apatient configured to be tracked by a navigation system. To accomplishthe accurate planning, tracking and navigation of surgical instruments,tools and/or medical devices during a surgical procedure utilizingsurgical navigation, surgeons often couple “tracking arrays” to thesurgical markers. These tracking arrays allow the surgeons to track thephysical location of these surgical components, as well as the patient'sbones during the surgery. By knowing the physical location of thetracking array, software associated with the tracking system canaccurately calculate the position of the tracked component relative to asurgical plan image. However, the use of markers that are screwed ontothe bone creates a risk of fracture at the anchorage point and lengthensthe operating time. Also, the visibility of the markers (Line Of Sight)must be guaranteed during the entire operation, therefore these kinds ofsolutions are not robust to occlusions, and constrain the positioning ofthe surgical staff in relation to the surgical field. They multiply thenumber of steps and tools (markers, probe) and require a mechanicalpalpation of the anatomical structure to be matched with the digitalmodel. These constraints increase procedure time with tedious work andrequire the surgeon and the surgical staff to learn how to place themarkers and how the operation is carried out. Finally, the associatedoptical tracking system are cumbersome and expensive.

Furthermore, no purely optical tracking allows for a real-time controlloop. To reach this level of performance, mechanical tracking isrequired.

The second type of solution involves ionizing intraoperativevisualization solutions, such as fluoroscopy. During orthopedic andtrauma surgery, a fluoroscopy-based navigation system allows thetracking of surgical instruments and the superposition of their contouronto fluoroscopy images in real time. For fluoro-navigation, theinstruments used during the surgical intervention are equipped withmarkers coupled to a localization system.

However, this second type of solution also suffers from the samedisadvantages of the first one: lack of robustness to occlusions,constraining the surgical staff positioning with respect to the surgicalfield, multiplying the number of instruments needed, bringing additionalcumbersome equipment to the surgical room (C-arm devices). This secondsolution further implies the use of ionizing radiation, which is harmfulfor both the patient and the surrounding medical staff performing thesurgery.

SUMMARY

The present invention is a method allowing the alignment of at least oneplanned surgical action defined in a virtual environment to a physicalsurgical action in a real environment, more specifically an operatingtheatre for an orthopedic surgery.

The implementation of this invention relates, for example, to situationswhere a trajectory, position, displacement or other actions are firstdetermined by means of a digital simulation, for example a surgicalplanning simulation, in a virtual referential. In order to be usable ina real environment, it is necessary to transpose this trajectory,position, displacement or other actions into a real referential of thepatient, on which the actions are to be applied. To do this, multipletransformations must be determined to match the virtual referential tothe real referential of the patient.

Preferably, the invention relates to the automated control of a supportfor a bone's machining tool, in order to comply with surgical planning

Another example of application of the invention relates to augmentedreality, for controlling the position of an image superimposed on thearea of vision of an operator, for example a surgeon, either from anaugmented vision goggle or from a projector.

The present invention relates to a system for computer guided surgerycomprising a transposition of an action, planned in a virtualenvironment with respect to a virtual referential R_(P), to a physicalaction performed with a surgical tool in a real operating theatreenvironment for orthopedic surgery of a patient, said surgical tool isfixed to a kinematic chain comprising a sensor unit having at least onesensor configured to follow in real time a spatial configuration of thekinematic chain; said system comprising:

-   -   a reception module configured to receive at least one 3D image        acquired from at least one 3D imaging sensor; said 3D image        comprising at least one portion of a target anatomical structure        of the patient;    -   a calculation module configured to:        -   calculate a transformation ^(C)T_(P) between the virtual            referential R_(P) and a target referential R_(C) and a            transformation ^(C)T_(A) between an acquisition referential            of the 3D imaging sensor R_(A) and the target referential            R_(C) by registration of a digital model of the target with            the at least one portion of the target comprised in the 3D            image;        -   apply the transformation ^(C)T_(P) so as to register said            digital model of the target in the target referential R_(C)            so that each point comprised in the digital model of the            target has a known position in the target referential R_(C);        -   calculate a transformation ^(C)T_(O) between a referential            of the surgical tool R_(O) and the target referential R_(C);        -   apply the transformation ^(C)T_(O) to the referential of the            surgical tool R_(O) so as to know the position and spatial            orientation of the surgical tool in the target referential            R_(C); so as to know the position and spatial orientation of            said surgical tool in both the virtual referential R_(P) and            the target referential R_(C) in order to reproduce the            action planned in the virtual referential R_(P) in the            target referential R_(C).

According to one embodiment, the calculation module is furtherconfigured to calculate the transformation ^(C)T_(A) by:

-   -   defining a region of interest in the 3D image comprising said        target;    -   registering said region of interest comprising the target to the        digital model of the target so as to determine ^(C)T_(A).

According to one embodiment, defining a region of interest comprises anautomated detection of said region of interest by means of asegmentation algorithm.

According to one embodiment, the kinematic chain comprises at least onemechanical reference rigidly fixed to the target anatomical structureand the at least one 3D image comprises at least one portion of themechanical reference; the calculation module is further configured to:

-   -   calculate a transformation ^(O)T_(M) between the referential of        the surgical tool R_(O) and the referential of the mechanical        reference R_(M) using the data obtained from the sensor unit        comprised in the kinematic chain;    -   calculate a transformation ^(M)T_(A) between the referential of        the mechanical reference R_(M) and the acquisition referential        R_(A) by matching a digital model of the mechanical reference        with the at least one portion of the mechanical reference        comprised in the 3D image;    -   so that the ^(C)T_(O) transformation is obtained from the        combination of the transformations ^(O)T_(M), ^(M)T_(A) and        ^(C)T_(A) between the acquisition referential R_(A), the        mechanical reference referential R_(M) and the target        referential R_(C).

According to one embodiment wherein the target anatomical structure isfixed on the at least one mechanical reference, the system furthercomprises a correction module configured to track the movements of thetarget with respect to the surgical tool using the sensor unit of thekinematic chain and, so that whenever a deviation in the position and/orspatial orientation of the target is detected, the transposition of theplanned actions from the virtual environment to the real environment iscorrected for said deviation. This embodiment advantageously allows toavoid needing a visual tracking or a new registration procedure when thepatient moves.

According to one embodiment, the at least one 3D imaging sensor is fixedto the kinematic chain, the calculation module is further configured tocalculate a transformation ^(A)T_(O) between the referential of thesurgical tool R_(O) and the acquisition referential of the 3D imagingsensor R_(A) from data obtained from the sensor unit of the kinematicchain so that the ^(C)T_(O) transformation is obtained from thecombination of the transformation ^(A)T_(O) and the transformation^(C)T_(A) between the acquisition referential R_(A) and the targetreferential R_(C).

According to one embodiment, the acquisition of the 3D image received bythe reception module is carried out using at least two sensors and aprojector to carry out an acquisition by stereovision or structuredlight.

According to one embodiment, the 3D imaging sensor fixed on thekinematic chain moves along a known trajectory and multiple 3D imagesare acquired along the trajectory, the calculation module is furtherconfigured to jointly process multiple 3D images acquired along thetrajectory so as to use multiple 3D images for the registration with thedigital model of the target.

According to one embodiment, the reception module is further configuredto receive a thermal image, an ultrasound image, a multispectral image,an image on a microscopic scale and/or a monocular color image.

According to one embodiment when the 3D imaging sensor is fixed to thekinematic chain, the system further comprises a correction moduleconfigured to track the movements of the target with respect to thesurgical tool using the 3D imaging sensor and a visual trackingalgorithm, so that whenever so that whenever a deviation in the positionand/or spatial orientation of the target is detected, the transpositionof the planned actions from the virtual environment to the realenvironment is corrected for said deviation.

The present invention relates to a method for computer guided surgerycomprising a transposition of an action, planned in a virtualenvironment with respect to a virtual referential R_(P), to a physicalaction performed with a surgical tool in a real operating theatreenvironment for orthopedic surgery of a patient, said surgical tool isfixed to a kinematic chain comprising a sensor unit having at least onesensor configured to follow in real time a spatial configuration of thekinematic chain; said method comprising the following steps:

-   -   receiving of at least one 3D image acquired from at least one 3D        imaging sensor; said 3D image comprising at least one portion of        a target anatomical structure of the patient;    -   calculating a transformation ^(C)T_(P) between the virtual        referential R_(P) and a target referential R_(C) and a        transformation ^(C)T_(A) between an acquisition referential of        the 3D imaging sensor R_(A) and the target referential R_(C) by        a registration of a digital model of the target with the at        least one portion of the target comprised in the 3D image;    -   applying the transformation ^(C)T_(P) so as to register said        digital model of the target in the target referential R_(C) so        that each point comprised in the digital model of the target has        a known position in the target referential R_(C);    -   calculating a transformation ^(C)T_(O) between a referential of        the surgical tool R_(O) and the target referential R_(C);    -   applying the transformation ^(C)T_(O) to the referential of the        surgical tool R_(O) so as to know the position and spatial        orientation of the surgical tool in the target referential        R_(C);    -   so as to know the position and spatial orientation of said        surgical tool in both the virtual referential R_(P) and the        target referential R_(C) in order to reproduce the action        planned in the virtual referential R_(P) in the target        referential R_(C).

According to one embodiment, the calculation of transformation ^(C)T_(A)comprises the following steps:

-   -   defining a region of interest in the 3D image comprising said        target;    -   registering said region of interest comprising the target to the        digital model of the target so as to determine ^(C)T_(A).

According to one embodiment, the step of defining a region of interestcomprises an automated detection of said region of interest by means ofa segmentation algorithm.

According to one embodiment, the at least one 3D imaging sensor is fixedto the kinematic chain the method further comprises the steps ofcalculating a transformation ^(A)T_(O) between the referential of thesurgical tool R_(O) and the acquisition referential of the 3D imagingsensor R_(A) from data obtained from the sensor unit of the kinematicchain so that the ^(C)T_(O) transformation is obtained from thecombination of the transformation ^(A)T_(O) and the transformation^(C)T_(A) between the acquisition referential R_(A) and the targetreferential R_(C).

According to one embodiment, the kinematic chain comprises at least onemechanical reference rigidly fixed to the target anatomical structureand the at least one 3D image comprises at least one portion of themechanical reference; the method further comprises the steps of:

-   -   calculating a transformation ^(O)T_(M) between the referential        of the surgical tool R_(O) and the referential of the mechanical        reference R_(M) using the data obtained from the sensor unit        comprised in the kinematic chain;    -   calculating a transformation ^(M)T_(A) between the referential        of the mechanical reference R_(M) and the acquisition        referential R_(A) by matching a digital model of the mechanical        reference with the at least one portion of the mechanical        reference comprised in the 3D image;    -   so that the ^(C)T_(O) transformation is obtained from the        combination of the transformations ^(O)T_(M), ^(M)T_(A) and        ^(C)T_(A) between the acquisition referential R_(A), the        mechanical reference referential R_(M) and the target        referential R_(C).

According to one embodiment wherein the mechanical reference is rigidlyfixed to the target anatomical structure, the movements of the targetwith respect to said surgical tool are tracked by the sensor unit of thekinematic chain, so that whenever a deviation in the position and/orspatial orientation of the target is detected, the transposition of theplanned actions from the virtual environment to the real environment iscorrected for said deviation.

According to one embodiment, said kinematic chain consists of adeformable structure comprising multiple rigid elements connected byjoints.

According to one embodiment, said kinematic chain further comprisessensors for measuring the forces applied to its elements.

According to one embodiment, the acquisition of the 3D image received bythe reception module is carried out using at least two sensors and aprojector to carry out an acquisition by stereovision or structuredlight.

According to one embodiment, the 3D imaging sensor fixed on thekinematic chain moves along a known trajectory and multiple 3D imagesare acquired along the trajectory, the method further comprises a stepof jointly processing multiple 3D images acquired along the trajectoryso as to use multiple 3D images for the registration with the digitalmodel of the target.

According to one embodiment, the method further comprises receiving athermal image, an ultrasound image, a multispectral image, an image on amicroscopic scale and/or a monocular color image.

According to one embodiment where the 3D imaging sensor is fixed on thekinematic chain, the movements of the target with respect to saidsurgical tool are tracked by the 3D imaging sensor and a visual trackingalgorithm, so that whenever a deviation in the position and/or spatialorientation of the target is detected, the transposition of the plannedactions from the virtual environment to the real environment iscorrected for said deviation.

According to one embodiment, the three-dimensional digital model of thetarget is generated from computed tomography images or MRI images.

According to one embodiment, the three-dimensional digital model of thetarget is generated using 2D X-ray radiographies comprising the target,a statistical shape model of the target and/or the 3D image acquiredintraoperatively by the 3D imaging sensor.

According to one embodiment, the three-dimensional digital model of thetarget is digitally modified to simulate measurement noise or thepresence of cartilage, said modifications being calculated from trainingdata or biomechanical simulation data.

According to one embodiment, the action of matching a digital model ofthe target with the at least one portion of the target comprised in the3D image is a non-rigid transformation.

The present invention further relates to a computer program productcomprising instructions which, when the program is executed by acomputer, cause the computer to carry out the steps of the methodaccording to any one of the embodiments described herein.

The present invention also relates to a computer-readable storage mediumcomprising instructions which, when executed by a computer, cause thecomputer to carry out the steps of the method according to any one ofthe embodiments described herein.

As described above, one embodiment of the present invention relates to amethod for computer guided surgery using a kinematic chain comprising atleast one mechanical reference rigidly fixed to the target anatomicalstructure. The following paragraphs relates to this specific embodiment.

The method implements a mechanical reference kinematically linkedthrough the kinematic chain with the surgical tool: during the entireoperation, the position and orientation of the surgical tool in relationto the mechanical reference are known thanks to position and/ordisplacement sensors of the components of the kinematic chain. On theother hand, the mechanical reference is rigidly linked to the boneelement to be machined (i.e. target). During the operation, theinvention uses depth data (also called 3D images) to perform aregistration of the bone element with planning data obtainedpreoperatively or intraoperatively (at the beginning of the operation,after soft tissues incision but before bone resections), said planningdata comprising the digital model of the target and the planned surgicalaction to be performed during the orthopedic surgery. In this way, theposition and orientation of the mechanical reference with respect to thetarget is determined. The registration process and the knowledge of thekinematic chain's geometry make it possible to determine the trajectoryof the surgical tool in the reference frame of the target. The positionand orientation of the surgical tool relative to the bone element isknown at all times and its path can be corrected in the event, forinstance, of patient movement, in order to comply with the previouslyestablished surgical planning

The present invention overcomes the disadvantages of the prior art inthat it does not require ionizing imaging or multiple optical markers toenable a surgical tool holder to be controlled in order to perform theactions comprised in the surgical planning Both being potentiallyharmful and time consuming, the present invention presents a directhealth benefit both for medical teams and patients. It also offersadvantages in terms of procedure length and ease of use for the surgeon.

The present invention relates to a method comprising the followingsteps.

A first step of acquisition of at least one 3D image from a 3D imagesensor(s) of a scene including:

-   -   at least one portion of a target (i.e. the bone element to be        machined);    -   at least one portion of the mechanical reference kinematically        linked to said tool.

A second step of mapping said 3D image to a digital model of the targetby applying image processing algorithms so as to determine thetransformation ^(C)T_(A) which defines the position and spatialorientation of the acquisition referential R_(A) with respect to thereferential of the target R_(C).

A third step of mapping said 3D image to a digital model of themechanical reference, making it possible to determine the transformation^(M)T_(A) which defines the position and orientation of the acquisitionreferential of the 3D image sensor R_(A) with respect to the referentialof the mechanical reference R_(M).

A fourth step of calculating the transformation matrix between thetarget referential R_(C) and the mechanical reference referential R_(M)using the ^(C)T_(A) and ^(M)T_(A) transformation matrices.

A fifth step of transposition of the initial planning into the targetreferential R_(C).

In parallel, the method comprises steps of real-time acquisition of thepositions, spatial orientation and/or displacements of the elements ofthe kinematic chain linking said mechanical reference to said surgicaltool during said physical action using the surgical tool.

Advantageously, the present method does not rely on real-time opticalmonitoring of the scene. It therefore avoids the problems associatedwith standard surgical navigation systems such as the multiplicity ofmarkers and the loss of tracking in the event of occlusion of theoperating area. Indeed, once the planning data are known in the targetreferential R_(C), the sensors of the kinematic chain provide knowledgeof the position and orientation of the surgical tool in relation to thetarget to be calculated at any time.

According to a variant, the said second and third mapping steps arereplaced by the following steps.

A region of interest extraction step consisting of extracting a firstregion of interest corresponding to said target and a second region ofinterest corresponding to said mechanical reference to determine:

-   -   a first subset of said 3D digital image associated with said        target;    -   a second subset of said 3D digital image associated with said        mechanical reference.

The extraction of the regions of interest can be performed automaticallythrough segmentation algorithms

A step of mapping said first subset associated with said target to thedigital model of said target to determine the transformation ^(C)T_(A).

A step of mapping said subset associated with said mechanical referenceto the digital model of said mechanical reference to determine thetransformation ^(M)T_(A).

According to variations of the embodiment, taken individually or in atechnically realistic combination, the invention also relates to thefollowing additional features:

-   -   the kinematic chain consists of a rigid deformable structure        comprising sensors measuring the relative positions of its        constituents;    -   said first step of acquiring a 3D image is carried out by        textured 3D image acquisition using at least two cameras and a        projector to carry out an acquisition by stereovision;    -   said first step of acquiring a 3D image is carried out by        textured 3D image acquisition using at least two sensors and a        projector to carry out an acquisition by structured light;    -   said first acquisition step further comprises acquisition of an        RGB-D image;    -   said first acquisition step further comprises acquisition of a        thermal image;    -   said first acquisition step further includes ultrasonic image        acquisition;    -   said first acquisition step further comprises acquisition of a        multispectral image;    -   said first acquisition step further comprises an acquisition of        an image on a microscopic scale;    -   said first acquisition step further includes acquisition of a        monocular color image;    -   the movements of said surgical tool with respect to the target        are tracked by means of position and/or displacement sensors of        the elements of the kinematic chain between the target and said        surgical tool, making it possible to correct the transposition        of the actions planned in a virtual environment to the real        environment;    -   said three-dimensional digital model of the target has been        generated from scanner or MRI images;    -   said three-dimensional digital model of the target has been        digitally modified to simulate measurement noise or the presence        of cartilage, said modifications being calculated from training        data or biomechanical simulation data;

Advantageously, said digital resetting processing of the second stepmaking possible to determine the transformation ^(C)T_(A) which definesthe position and orientation of the acquisition referential R_(A) withrespect to the target referential R_(C) is non-rigid.

Definitions

In the present invention, the following terms have the followingmeanings:

-   -   “3D sensor” or “3D camera” or “Depth Camera” or “3D scanner”        refers to a 3D sensor is a system for acquiring topological data        of a real scene in 3 dimensions. These topological data are        recorded in the form of a point cloud, and/or a depth map.    -   Multiple acquisition technics allow to obtain these topological        data for example:        -   technics based on the measure of waves propagation time such            as ultrasound or light (LIDAR, Time-of-Flight);        -   stereoscopic camera or sensor, which is a type of camera            with two or more lenses with a separate image sensor or film            frame for each lens. This allows the camera to simulate            human binocular vision, and therefore gives it the ability            to capture three-dimensional images;        -   technics based on light deformation, such as            structured-light 3D scanners which project a pattern of            light on an object and look at the deformation of the            pattern on the object. The advantage of structured-light 3D            scanners is speed and precision. Instead of scanning one            point at a time, structured light scanners scan multiple            points or the entire field of view at once. Scanning an            entire field of view in a fraction of a second reduces or            eliminates the problem of distortion from motion;        -   technics based on laser scanning for sample or scan a            surface using laser technology, such as hand-held laser or            time-of-flight 3D laser scanner;    -   These terms may as well refer to an RGB-D, color, multispectral,        or a thermal camera.    -   “Referential” refers to a coordinate system that uses one or        more numbers, or coordinates, to uniquely determine the position        of the points or other geometric elements on a manifold such as        Euclidean space.    -   “Tracking”, in computer vision, refers to the action of        following the position and spatial orientation of an object        between successive images of a stream. Marker-based tracking        relies on the use of a localization device associated to a        marker attached to the object of interest. Marker-less tracking        relies on extracting visual features from the object of interest        itself and matching them on a frame-to-frame basis.    -   “Registration” or “Matching” or “Pose estimation” refers to the        process of transforming different sets of data into one        coordinate system.    -   “Tridimensional digital model” refers to a three-dimensional        digital (or virtual) model being a virtual object in 3        dimensions. The position and orientation of the model is known        in the associated digital referential.    -   “Planning”, in the context of surgery, refers to a list of        actions to be performed during the different surgical phases.        This surgical planning may be obtained by means of a simulation        program carried out before the operation which uses a        3-dimensional digital model of the bone(s) of the patient that        are the target of the surgery. In the case of a knee        arthroplasty operation, for example, pre-operative planning will        consist of defining each of the machining planes and drilling        axes in relation to a three-dimensional model of the femur and        tibia.    -   “Pre-operative data” refers to images (or slices) of the patient        obtained by medical imaging (CT, MRI, PET, etc.). The        three-dimensional model of the patient is obtained by a        segmentation treatment of each image, followed by an        interpolation between the images.    -   “Intraoperative data” refers to data acquired during the        operation. This may include medical imaging (fluoroscopy,        ultrasound, etc.), three-dimensional data, color and temperature        information, information from proprioceptive sensors, force        feedback with respect to a surgical tool, etc.    -   “Machining” refers to the mechanical process of cutting or other        methods for removing material. The purpose of machining is to        modify the dimensions, precision, geometry and surface state of        all the surfaces of the finished element, in order to move from        a raw original state to a final state in accordance with a        predefined model.

DETAILED DESCRIPTION

The following detailed description will be better understood when readin conjunction with the drawings. For the purpose of illustrating, thesteps implemented by the system and the method are shown in thepreferred embodiments. It should be understood, however that the presentinvention is not limited to the precise arrangements, structures,features, embodiments, and aspect shown. The drawings are not drawn toscale and are not intended to limit the scope of the claims to theembodiments depicted. Accordingly, it should be understood that wherefeatures mentioned in the appended claims are followed by referencesigns, such signs are included solely for the purpose of enhancing theintelligibility of the claims and are in no way limiting on the scope ofthe claims.

Features and advantages of the invention will become apparent from thefollowing description of embodiments of a system, this description beinggiven merely by way of example and with reference to the appendeddrawings in which:

FIGS. 1 and 1B show a schematic view of the different referentialsystems defined in the present invention according to the embodimentwhere the kinematic chain comprises at least one mechanical reference.

FIG. 2 shows a schematic view of the step of the process where thetarget referential R_(C) and the virtual referential R_(P) are alignedto determine the ^(C)T_(A) transformation that defines the position andorientation of the acquisition referential R_(A) with respect to R_(C).^(C)T_(A) is a homogeneous transformation matrix composed of a ^(C)R_(A)rotation matrix and a ^(C)T_(A) translation vector. It defines theposition of target referential R_(C) with respect to acquisitionreferential R_(A).

FIG. 3 shows the fourth step according to one embodiment of theinvention, where the position and spatial orientation of the mechanicalreference referential R_(M) relative to acquisition referential R_(A) isdetermined (transformation matrix ^(M)T_(A)). ^(M)T_(A) is a homogeneoustransformation matrix, composed of a rotation matrix ^(M)R_(A) and atranslation vector ^(M)T_(A). It defines the position and spatialorientation of mechanical reference referential R_(M) with respect toacquisition referential R_(A).

FIG. 4 represents the ^(C)T_(M) transformation between the mechanicalreference and the target is calculated by the composition of the^(A)T_(C) (inverse ^(C)T_(A) matrix) and ^(M)T_(A) matrices.

FIG. 5 corresponds to the sixth step of the process. The transformation^(O)T_(M), determined from the kinematic chain linking the mechanicalreference and the surgical tool, is combined with the transformations^(C)T_(M) to calculate ^(C)T_(O) (the position of the surgical tool inthe target and planning referential).

FIGS. 6 and 7 show two examples of how the system and the kinematicchain may be constructed, according to one embodiment where themechanical reference is kinematically linked to the surgical tool. Ahuman-machine interface, represented here by the screen 60, providesvisual feedback on the individual process steps.

FIG. 8 shows some machining planes P₁, P₂, P₃, P₄, P₅, P₆ for a femur Fand a tibia T in view of the implantation of a femoral knee implant I intotal knee arthroscopy surgery.

FIG. 9 shows an example of the system of the present invention and thekinematic chain comprising the surgical tool at one end, the 3D imagingsensor having a known position with respect to the patient.

FIG. 10 schematically represent the step of calculating a transformation^(C)T_(P) between the virtual referential R_(P) and a target referentialR_(C) and a transformation ^(C)T_(A) between an acquisition referentialof the 3D imaging sensor R_(A) and the target referential R_(C).

FIG. 11 schematically represent the step of calculating a transformation^(C)T_(O) between a referential of the surgical tool R_(O) and thetarget referential R_(C).

FIG. 12 is a schematic representation the embodiment where the at leastone 3D imaging sensor is fixed onto the kinematic chain. In this figureare as well illustrated the transformation ^(A)T_(O) between thereferential of the surgical tool R_(O) and the acquisition referentialof the 3D imaging sensor R_(A) and the transformation ^(C)T_(A) betweenthe acquisition referential R_(A) and the target referential R_(C),which are used to calculate the transformation ^(C)T_(O).

While various embodiments have been described and illustrated, thedetailed description is not to be construed as being limited hereto.Various modifications can be made to the embodiments by those skilled inthe art without departing from the true spirit and scope of thedisclosure as defined by the claims.

The purpose of the present invention is to match, in an uninterruptedand real-time manner, the pre-operative planning data with the actualprocedure taking place in the surgical theater.

The pre-operative surgical planning comprises a 3D digital model of atleast a part of the patient, notably comprising the target bone element,and an ordered set of geometrical equations characterizing the machiningactions on the patient's target bone elements. In a preferredembodiment, the pre-operative surgical planning comprises the machiningplan corresponding to each surgical action. With the term pre-operativesurgical planning it is understood that the surgical actions of thesurgical planning have been defined using planning data obtained beforethe surgery (pre-operatively) or intraoperatively during the first phaseof the surgery when the target bone element is exposed but before thebeginning of the surgical action on the target bone.

The digital model of the target may be generated from medical imagesacquired prior to the operation. It can be modified to take into accountelements that are not visible in the medical images, for examplecartilage that is not visible in CT scan images. In this case, themodifications are generated from training data or biomechanicalsimulation data. The digital model can also be generated fromstatistical models or abacuses and patient data associated or not withthe preoperative medical images. In addition, the digital model can beadapted taking into account data acquired during the operation.

Planning data for the determination of the pre-operative surgicalplanning are acquired in a place, time and with a position of thepatient with respect to the imaging means that can be completelyindependent from the ones of the surgery. However, the digital data ofthe digital planning model must be brought closer to physical reality inorder to allow a control of the movements of the real machining tool asfunction the digital planning data.

For the rest of the description, the following references will beconsidered:

-   -   the planning or virtual referential R_(P) of digital planning        images and surgical actions of the surgical planning stored in a        computer's memory. The bone element to be machined in the        digital model of the target, in the described example a femur        10, has a known position and spatial orientation in this virtual        referential R_(P). The surgical planning data and surgical        actions (e.g. the position of the cutting planes 11 or the        drilling axes) are known in the same referential R_(P);    -   the target referential R_(C) corresponds to the physical        coordinate system of the bone element to be machined (i.e.        target), in this case the surface of the femoral head;    -   the surgical tool referential R_(O) corresponds to the physical        coordinate system of the surgical tool 20;    -   the acquisition referential R_(A) corresponds to the 3D imaging        sensor 30 coordinate system in which the data acquired during        the operation are represented    -   the mechanical referential R_(M) corresponds to the physical        coordinate system of the mechanical reference 40 kinematically        linked to the surgical tool 20 through the element of the        kinematic chain. The position and spatial orientation of the        surgical tool 20 relative to the mechanical reference 40 are        known by the sensors of the kinematic chain providing signals        representative of the angular and/or linear displacements of the        surgical tool 20 with respect to the mechanical reference 40.

The positioning of the bone target 10, the kinematic chain 70, thesurgical tool 20, the 3D imaging sensor 30 and their respectivereferential are represented in FIG. 9.

The present method aims to accurately guide a surgical tool 20 movablyfixed to a kinematic chain 70. Said surgical tool 20 may be for examplea machining tool.

In the present invention the term kinematic chain refers to an ensembleof rigid elements connected by joints to constrain or provide motion ina desired way.

According to one embodiment, said kinematic chain consists of adeformable structure comprising multiple rigid elements connected byjoints.

According to the present invention, the kinematic chain comprises asensor unit having at least one sensor configured to follow in real timea spatial configuration of said kinematic chain.

The sensors of the sensor unit may be encoders or inertial unitscomprising accelerometer and/or gyroscopes.

According to one embodiment, said kinematic chain further comprisessensors for measuring the forces applied to its elements.

In one embodiment, the first step of the present method comprisesreceiving at least one 3D image acquired from at least one 3D imagingsensor 30 wherein the 3D image is acquired so as to comprise at leastone portion of a target anatomical structure 10 of the patient.

The 3D image obtained from the 3D imaging sensor 30 of the presentinvention comprises information of the distance between each point ofthe scene acquired in the image and the 3D imaging sensor 30. Therefore,the raw 3D image obtained by the 3D imaging sensor 30 is a so called adepth map, or depth image that may be presented under the form of abidimensional array representing a grey level image or a RGB image,wherein the size of the array depends on the camera type and sensordimensions.

According to one embodiment, the acquisition of the 3D image is carriedout using at least two cameras and a projector to carry out anacquisition by stereovision or structured light.

The use of a 3D imaging sensor advantageously allows to obtaininformation on the morphology of the bone surface in an easy and fastway since one image allows to capture all the surgical field, withoutcoming into contact with the patient (as for palpation techniques).

According to one embodiment, the method further comprises apre-processing step implementing noise reduction algorithms

According to one embodiment, the at least one 3D imaging sensor 30 has afixed position with respect to the target 10 in the surgical theatre. Inthis embodiment, the 3D imaging sensor 30 is independent from thekinematic chain (i.e. the 3D imaging sensor is not fixed to thekinematic chain). In one example, the 3D imaging sensor 30 is fixed on awall of the surgical theatre or is positioned on a tripod or fixed bymeans of an articulated arm. In case the 3D imaging sensor 30 isdisplaced so as to capture multiple 3D images, an inertial measurementunit (IMU) fixed to the 3D imaging sensor 30 can measure its relativemovement and determine the motion trajectory.

According to an alternative embodiment, the at least one 3D imagingsensor 30 is fixed to the kinematic chain as illustrated in FIG. 12.This advantageously allows to have access at all times during thesurgery to the relative position of the surgical tool 20 with respect tothe 3D imaging sensor 30.

When the at least one 3D imaging sensor 30 is fixed to the kinematicchain 70 then the 3D imaging sensor moves along a known trajectory, the3D imaging sensor 30 may acquire multiple 3D images along thistrajectory.

According to one embodiment, the first step further comprises receivinga thermal image, an ultrasound image, a multispectral image, an image ona microscopic scale and/or a color image.

According to one embodiment, the method comprises retrieving from acomputer readable storage medium, a server or the like a digital modelof the target bone to be treated during the surgery using the surgicaltool 20. Said digital model is a three-dimensional virtualrepresentation of the target bone 10.

In one embodiment, the three-dimensional digital model of the target isgenerated using imaging data acquired using computed tomography or MRIsystems. Other imaging techniques may be as well used such as X-rays,fluoroscopy, ultrasound or other imaging means. In this case, thethree-dimensional digital model is obtained previous to the surgery.

In one embodiment, the three-dimensional digital model of the target isgenerated using 2D X-ray radiographies comprising the target, astatistical shape model of the target and/or the 3D image acquiredintraoperatively by the 3D imaging sensor 30.

This embodiment advantageously allows to generate a three-dimensionalmodel even when the 3D imaging data (i.e. computed tomography or MRI)are not available.

In one embodiment, the three-dimensional digital model of the target ismodified to simulate measurement noise or the presence of cartilage.Said modifications may be calculated from training data or biomechanicalsimulation data.

According to one embodiment illustrated in FIGS. 2 and 10, a furtherstep of the method comprises calculating a transformation ^(C)T_(P)between the virtual referential R_(P) and a target referential R_(C) anda transformation ^(C)T_(A) between an acquisition referential of the 3Dimaging sensor R_(A) and the target referential R_(C) by a registrationof the digital model of the target with the at least one portion of thetarget 10 comprised in the 3D image.

3D registration consisting in finding the transformation between two 3Dmodels of the same object such that their overlapping areas match aswell as possible. This can be performed by an algorithm that iterativelyaligns the two models by alternating a matching step associating eachpoint of the intraoperative image to its nearest neighbor in thepreoperative model, and a transformation estimation step whichtransforms the intraoperative model as to best fit the estimatedmatches. This process is repeated until the distance between each pointof the intraoperative and preoperative model is minimized, below athreshold value.

This step advantageously allows to know the transformation that alignsthe virtual referential R_(P) to the target referential R_(C) in thesurgical theatre. Furthermore, the use of a digital model of the boneand its registration to at least one 3D image of target in the surgicalfield allows to know the transformation between the between the virtualreferential R_(P) and a target referential R_(C) independently from anyexternal marker attached to the patient.

In one embodiment, the registration of the digital model of the targetwith the at least one portion of the target comprised in the 3D image isobtained from a rigid transformation.

Alternatively, the registration of the digital model of the target withthe at least one portion of the target comprised in the 3D image may bea non-rigid transformation.

Advantageously, this embodiment allows to adapt the shape of the digitalmodel of the target obtained from pre-acquired images to the 3D imageacquired during the surgery.

According to one embodiment, the step of calculating the transformation^(C)TA comprises a first step of defining a region of interest in the 3Dimage which comprises at least a portion of the target.

According to one embodiment, the step of defining a region of interestcomprises an automated detection of said region of interest by means ofa segmentation algorithm.

Alternatively, an operator may provide as input to the methodinformation comprising a manual delineation of the contour of the regionof interest of the target.

This region of interest comprising the target is then registered to thedigital model of the target so as to determine ^(C)T_(A).

In one embodiment, the method further comprises a step of applying thetransformation ^(C)T_(P) so as to register said digital model of thetarget in the target referential R_(C) so that each point comprised inthe digital model of the target has a known position in the targetreferential R_(C). This step advantageously allows to align the virtualreferential to which are associate the digital model of the target andthe actions of the surgical planning to the target referential R_(C) inthe surgical theatre.

In one embodiment illustrated in FIG. 11, the method comprises a step ofcalculating a transformation ^(C)T_(O) between a referential of thesurgical tool R_(O) and the target referential R_(C).

The method then may implement a step of applying the transformation^(C)T_(O) to the referential of the surgical tool R_(O) so as to knowthe position and spatial orientation of the surgical tool 20 in thetarget referential R_(C). This final step allows to know the positionand spatial orientation of said surgical tool 20 in both the virtualreferential R_(P) and the target referential R_(C) in order to reproducethe action planned in the virtual referential R_(P) in the targetreferential R_(C).

When the position and spatial orientation of said surgical tool 20 inboth the virtual referential R_(P) and the target referential R_(C) isknown, it is possible to make use of the kinematic chain carrying thesaid surgical tool 20 to guide the performance of the actions planned inthe initial surgical planning. However, it may happen that during theperformance of these actions the spatial orientation and position of thetarget is modified, for example by a movement of the target by one ofthe members of the medical staff. This would result into a mismatchbetween the virtual referential R_(P) and the target referential R_(C)and therefore the surgical tool 20 executing the planned action withreference to the virtual referential R_(P) would be in a wrong position.

In order to prevent this undesired situation, the movements of thetarget 10 with respect to said surgical tool 20 may be tracked so thatwhenever so that whenever a deviation in the position and spatialorientation of the target is detected, the registration of the plannedactions from the virtual referential R_(P) and the target referentialR_(C) is promptly corrected for said deviation.

According to one embodiment where the 3D imaging sensor 30 is fixed tothe kinematic chain 70 as shown in FIG. 12, the method is configured tocalculate a transformation ^(A)T_(O) between the referential of thesurgical tool R_(O) and the acquisition referential of the 3D imagingsensor R_(A) from data obtained from the sensor unit of the kinematicchain and by combining said transformation ^(A)T_(O) to thetransformation ^(C)T_(A) so as to obtain the ^(C)T_(O) transformationbetween the surgical tool R_(O) and the target referential R_(C).

In the case where the 3D imaging sensor 30 is fixed to the kinematicchain 70 as shown in FIG. 12, according to one embodiment, the movementsof the target 10 with respect to said surgical tool 20 are computedthrough a visual tracking algorithm using as input the live 3D imagescaptured by the 3D imaging sensor 30 for a continuous pose (i.e. spatialorientation and position) estimation of the target 10 with respect tothe 3D imaging sensor 30. The relative motion of the 3D imaging sensor30 and target 10 is computed by performing a frame-to-frameregistration, namely by registering the current 3D image (timestep i)with the previous one (timestep i−1). Since the 3D imaging sensor 30 ismechanically linked to the surgical tool 20, the relative motionestimated from the visual tracking should match the relative motioncomputed from the sensor units of the kinematic chain 70. If this is notthe case, it means that the target 10 has moved and the registration ofthe planned actions from the virtual referential R_(P) and the targetreferential R_(C) (first three steps of the present method) needs to beperformed again to correct the deviations.

According to the alternative embodiment where the kinematic chain 70 isindependent from the 3D imaging sensor 30, the kinematic chain 70comprises at least one mechanical reference 40 rigidly fixed to thetarget anatomical structure 10. According to this embodiment representedin FIGS. 1 and 1B, the at least one 3D image must comprise at least oneportion of the mechanical reference.

According to this embodiment shown in FIG. 3, the method furthercomprises the steps of:

-   -   calculating a transformation ^(O)T_(M) between the referential        of the surgical tool R_(O) and the referential of the mechanical        reference R_(M) using the data obtained from the sensor unit        comprised in the kinematic chain 70;    -   calculating a transformation ^(M)T_(A) between the referential        of the mechanical reference R_(M) and the acquisition        referential R_(A) by matching a digital model of the mechanical        reference with the at least one portion of the mechanical        reference comprised in the at least one 3D image;    -   so that the ^(C)T_(O) transformation is obtained from the        combination of the transformations ^(O)T_(M), ^(M)T_(A) and        ^(C)T_(A) between the acquisition referential R_(A), the        mechanical reference referential R_(M) and the target        referential R_(C).

When the mechanical reference 40 is fixed to the target 10 as shown inFIG. 1, 1B and 7, according to one embodiment, the movements of thetarget 10 with respect to said surgical tool 20 are tracked by thesensor unit of the kinematic chain 70 so that whenever a deviation inthe position and spatial orientation of the target 10 is detected, theregistration of the planned actions from the virtual referential R_(P)and the target referential R_(C) is promptly corrected for saiddeviation. Indeed, since the mechanical reference 40 is rigidly fixed tothe target anatomical structure 10 while being part of the kinematicchain 70, it is possible to detect all movements of the target 10 withrespect to said surgical tool 20 using the information acquired from thesensor unit. This correction may consist in the calculation of acorrection transformation ^(Cnew)T_(P) between the new targetreferential R_(C) and virtual referential R_(P) so that each pointcomprised in the digital model of the target has a known position in thenew target referential R_(C). with mechanical reference

The specific embodiments concerning the steps of the method to beimplement when a kinematic chain 70 comprising at least one mechanicalreference 40 is used, are described in details in the followingparagraphs.

As an example, the surgical tool 20 is associated with a tool holderconnected to a mechanical reference, such as a gripper or a screw,mechanically fixed to the bone to be machined. The mechanical connectionbetween this tool support and the mechanical reference is provided by akinematic chain having deformable structure comprising at least twoelements, for example an articulated assembly comprising multiplesarticulated elements and sensor unit having at least one sensorconfigured provide in real-time a signal as a function of the relativespatial orientation and position of the articulated elements. Thissensor unit thus provide digital information making it possible todetermine in real time the position in space of the active end of thesurgical tool with respect to a fixed point of the element fixed on thebone.

In addition, intraoperative data acquisition and their matching with thedigital surgical planning allows the positioning of the mechanicalreference fixed on the bone to be known in relation to the bone elementsurface to be machined. A known trajectory can be transposed into thereal world by simulation in a virtual referential, for example to ensuremachining tool guidance, control of its movements, or position controlin relation to a predetermined position during surgical planning

First Step

For this purpose, a 3D image is acquired by a 3D imaging sensor 30, forexample a camera, whose field of view encompasses part of the operatingfield in which is comprised at least part of the surface to be machined10 (for example a femur) and part of the mechanical reference 40. Theresult of the acquisition can be displayed on a screen 60.

The acquisition can be carried out by a 3D camera, a pair of cameras toacquire images in active stereovision, a 3D scanner or a LIDAR toprovide a three-dimensional image of type (x, y, z; a) or a depth map, apoint cloud, a designating a parameter such as color or intensity.

3D Image Acquisition of the Operation Scene

A textured 3D digitizing solution uses two or more calibrated camerasand a projector to perform stereovision acquisition and phase-shiftstructured light acquisition for an accurate 3D reconstruction of thesurgical area. The proposed solution integrates a spatio-temporalsuper-resolution scheme, with non-rigid 3D registration, to correct the3D information and complete the scanned view.

The structured light is encoded by time multiplexing. Two sinusoidalpatterns in phase opposition and a third white pattern are successivelyprojected onto the operative scene. A 2D sampling is first applied foreach camera separately to locate the fringe intersection points. Anon-dense 3D model of the scene is then estimated for each pair ofcameras by stereo matching between the obtained primitives and opticaltriangulation. The spatial resolution of this model depends on thenumber of fringes forming the pattern used. A dense 3D model is thenobtained by estimating the phase information of the points locatedinside the fringes, separately for each camera-projector pair used.Conventional phase-shift structured light-based approaches requireoff-line calibration of the projector with the cameras and a phaseunwinding step.

Super-resolution in space and time makes it possible to complete andcorrect 3D models of the observed scene, since 3D scanning can generatedistortions and artifacts caused mainly by occultations, by a variationin the position, or even by light reflection on the acquisition surface.Thus, by using on the one hand the different 3D models provided by allcamera pairs and on the other hand the 3D frame calculated at time t−1,a high-resolution 3D model corrected at time t is obtained. Thespace-time super-resolution is provided by a first 3D matching stepfollowed by a merging and denoising step. A non-rigid 3D matchingapproach allows to deal with a possible distortion or deformation of thenon-rigid observed area. A mesh of the obtained 3D point cloud and atexture plating allow to finalize the textured 3D frame of the instantt.

The result of this first step is the recording in the memory of thecomputer of a 3D image of the area containing both a visible part of thebone 50 and a visible part of the reference 40, in the form of a pointcloud each defined by a luminous intensity, the colors and thecoordinates (x, y, z) in the acquisition referential R_(A).

Additional Imaging Modalities

A particular solution is to acquire additional images of differentnature, in the same acquisition referential R_(A), or in a referentialcalibrated with R_(A) to have additional information. Calibration of theadditional modality requires the use of a geometric test pattern visiblefrom different angles of view by both the 3D imaging sensor 30 and theadditional modality. The resulting image pairs are processed andrescaled to derive the calibration matrix.

The additional image can for example be a thermal image produced by athermal camera that captures the operating field with an orientation anddistance close to that of the 3D imaging sensor 30. This imageadvantageously makes it easier to distinguish between the patienttissues and the surgical tool.

It can also be an acquisition by a color camera, an ultrasonic probe ora multispectral sensor.

Step 2: Extraction of Areas of Interest

The following processing step consists of exploiting the at least one 3Dimage recorded during the acquisition step to isolate the portion of theimage corresponding to the target 10 (femur), and the portion of theimage corresponding to the mechanical reference 40.

To do this, the said 3D digital image of the entire scene, and the imageobtained by said additional imagery if present, is processed byalgorithms for characterizing the subsets of the depth map or the pointcloud.

The result of this processing will be a segmentation or classification:

-   -   with a first indicator (label) associated with the mechanical        reference corresponding to a first subset of points of the 3D        image,    -   with a second indicator (label) associated with the target        (femur), corresponding to a second subset of points in the 3D        image,    -   with a third indicator (label) of the background (non-relevant        subset of the image).

This processing step will be carried out by successive contour, colorand in alternative using trained classifiers, or by artificialintelligence, or by taking into account the geometrical a priori of thepositioning of the mechanical reference and the target in relation tothe acquisition system.

This step of extracting areas of interest is optional if the matchingalgorithms applied during the third and fourth steps are sufficientlyrobust to outliers.

Third step: Matching the Physical Reference Frame Linked to the Targetwith the Acquisition Reference Frame.

The third step consists in matching the target referential R_(C)associated to said physical target with the acquisition referentialR_(A), by a registration processing between:

-   -   the subset of the three-dimensional digital image associated        with the target, determined in the previous steps, and    -   the three-dimensional digital model of the target recorded with        the planning data.

This processing consists in determining the ^(C)T_(A) transformation,giving the position and orientation of the acquisition referential R_(A)with respect to target referential R_(C), as shown in FIG. 2.

This processing uses registration techniques to find an underlyingdeformation common to two geometric structures of the same nature,allowing them to be linked, i.e. attempting to describe the secondstructure as being obtained from the first by applying a spatialtransformation.

The man skilled in the art knows said matching techniques based on theprior extraction of characteristic points, from which deformations areinduced, or the exploitation of geometrical structures derived from theoriginal images: points, portions of curves or surfaces obtained bysegmentation, insofar as these capture the essential information of theimages, whether geometrical—points or lines of strong curvature—oranatomical.

A suitable technique is based on point-to-point registration by means ofa process of estimating an optimal transformation between two sets ofdata, such that their overlapping areas match as well as possible. Thiscan be performed by an algorithm that iteratively aligns the two modelsby alternating a matching step associating each point of theintraoperative image to its nearest neighbor in the preoperative model,and a transformation estimation step which transforms the intraoperativecloud as to best fit the estimated matches. This process is repeateduntil the distance between each point of the intraoperative andpreoperative model is minimized, below a threshold value.

Registration is said to be rigid if the geometric transformationincludes rotation and translation. The registration is said to benon-rigid if the geometric transformation is of higher order(polynomial, splines . . . ) or if the transformation is not parametric.

In the context of the present invention, a rigid registration isgenerally sufficient to calculate the transformation matrix from thetarget referential to the virtual referential R_(P).

Fourth Step: Mapping of the Physical Reference Frame Linked to theMechanical Reference to the Acquisition Frame.

The fourth step consists in matching the mechanical referencereferential R_(M) associated to the said mechanical reference with theacquisition referential R_(A), by a registration processing between:

-   -   the 3D image of said mechanical reference, and    -   the three-dimensional digital model of said mechanical        reference.

The same type of rigid registration processing is applied using thesubset of the 3D image points corresponding to the mechanical reference,and its digital representation in the computer memory.

The result of this processing shown in FIG. 3 is used to determine the^(M)T_(A) transformation, giving the position and orientation of themechanical referential RM relative to acquisition referential R_(A).

Fifth Step: Transformation between Target Referential R_(C) andMechanical Referential R_(M)

The fifth step of the process shown in FIG. 4 consists in calculatingthe transformation between the target referential R_(C) and themechanical referential R_(M). Knowing the ^(C)T_(A) and ^(M)T_(A)matrices (shown in FIGS. 2 and 3) thanks to the two previous steps, itis possible to deduce the transformation matrix ^(M)T_(C) expressing therelation between the target referential R_(C) and the mechanicalreferential R_(M).

Sixth Step: Moving from Initial Planning to Physical Planning

The sixth step represented in FIG. 5 relates to the transposition of theplanned action of the initial surgical planning to physical actionsperformed in the target referential R_(C) by positioning the tool'ssupport according to the digital planning data thus transposed. The^(O)T_(M) transformation, determined using the data obtained from thesensors of the kinematic chain comprising the mechanical reference andthe machining tool, is combined with ^(M)T_(C) to calculate thetransformation ^(C)T_(O) allowing to compute the machining tool positionin the target referential, and thus transpose the planning data to thereal environment. The knowledge of transformation ^(C)T_(O) then enablesthe correction of the machining tool trajectory according to themovements of the target during the intervention.

Surgical planning data, transposed on intraoperative images, can bedisplayed on a screen 60 in the operating theatre. This provides thesurgeon with visual feedback on the progress of the procedure inrelation to the schedule.

According to one embodiment steps 2 and 3 are realized without passingthrough a subset extraction step, by using a single processing using analgorithm robust to outliers, e.g. Random Sample Consensus (RANSAC).

FIG. 8 provides an example of an anatomical structure A which isclassically well known for regularly needing surgery, is the knee joint.As known per se, the knee joint includes three bones, the femur F, thetibia T and the patella. (We will intentionally exclude the patella,from this description for it adds no explanatory value) The examplesdescribed in the present specification relate therefore to the field oforthopedic surgery and more specifically to the preparation of a femur Fand a tibia T for the implantation of a femoral knee implant I.

This preparation according to this example includes a succession ofwell-known steps, each step being the machining of one of the bones F orT (bone cut using traditionally an oscillating saw) according to a givenpre-calculated machining plan P₁, P₂, P₃, P₄, P₅, P₆ (see FIG. 8)comprised in the surgical planning Those machining steps are well-knownper se and they usually take place in the same order, depending on thestrategy adopted by the operators (surgeons). On FIG. 8, each machiningplan P₁, P₂, P₃, P₄, P₅, P₆ is numbered in the generally admittedchronological sequence. Those machining plans P₁, P₂, P₃, P₄, P₅, P₆ areclassically determined by a pre-operative surgical planning Apre-operative surgical planning is only valid for one given patient forone given surgery for one given type of implant (size, design, brand,etc.). Each patient (and each surgery) gets a personalized pre-operativesurgical planning Therefore, the machining plans P₁, P₂, P₃, P₄, P₅, P₆slightly change for each surgery. The usual first step of thepre-operative surgical planning, is to establish a 3D digital model ofthe target bones F, T. One way to obtain such as 3D digital bones F, Tmodel is to use medical imaging such as computed tomography, X-rays,MRI, fluoroscopy, ultrasound or other imaging means. X-ray or scanner,or even MRI, acquisitions are usually made during full weight-bearing,with typically a frontal (also named coronal or anteroposterior) view, alateral (or profile) view with the knee in full extension and/or at20°-30° of flexion, a long-leg view, including the lower limb from thefemoral head to the ankle joint and lastly a view of the kneecap at 30°flexion, also called skyline view. From these images it is possible tobuild a digital model of the bones F, T to be machined during theoperation. A particular knee implant I is then selected based on ananalysis of the 3D digital bones F, T model.

The present invention aims at allowing an accurate and safe machining ofthe bones F, T by means of a surgical device comprising a kinematicchain 70 with a surgical tool 20 as shown in FIG. 6, 7 or 9.

After being established, the F, T bones 3D digital model may be storedin a memory of a control unit of said surgical device.

In one example, the surgical device may include a 3D imaging sensor 30which its position is well known within the surgical device. This 3Dimaging sensor 30 allows the operator, in cooperation with the bones F,T model stored in the memory of the control unit. Once a F, T bones 3Ddigital model has been determined for a given patient, and stored insidethe memory of the control unit, the surgical device, it can be used forsurgery. Once the patient is correctly installed, the anatomicalstructure A to be seen and the surgery device correctly put in placewith regards to the patient, at least one 3D image of the anatomicalstructure A is taken. This 3D image is taken with the 3D imaging sensor30. The control unit of the surgical device may be configured to performthe step of the method of the present invention. This enables thecontrol unit to position the anatomical structure A with regards to the3D imaging sensor 30 and therefore to the surgery device. This thenenables the control unit to set the precise machining plans P₁, P₂, P₃,P₄, P₅, P₆ for this specific surgery within the target referentialR_(A).

The free surface of the bones F, T to be machined is limited and thereare therefore only a few areas where a surgical tool 20 can be put incontact with the bones F, T. This contact has to be as minimallyinvasive as possible in order to damage neither the bones F, T, nor thesurrounding soft tissue while ensuring a precise relative positioning ofthe surgical tool 20 relative to the bones F, T.

As shown in FIG. 6, 7 or 9, the surgical device aims at machining ananatomical structure A (in this case, a knee) of a patient positioned onan operation table. The patient is usually anesthetized and maintainedon the operation table by means of specific and well-known fixationmeans. In one embodiment illustrated in FIGS. 6 and 7, in addition, thepatient's limb in its whole is secured to the kinematic chain 70 of thesurgical device.

For example, in addition to the kinematic chain 70 and the surgical tool20, the surgical device may comprise a base unit aimed at being securedto the operation table and a mechanical reference 40 designed to securethe anatomical structure A. The surgical tool 20 may be configured to bedisplaced by the operator.

In one embodiment the system for computer guided surgery corresponds tothe control unit of said surgical device. Said control unit 80 can forexample be a computer. This control unit 80 may comprise a memory, areal time computing element, power supply, power converters, fusesand/or actuators. The control unit 80 may further comprise an operatorinterface 60 allowing an interaction between the control unit 80 and theoperator.

This operator interface 60 may be configured to

-   -   display the images acquired by the 3D sensor and the output of        steps one to three    -   display real time information such as the surgical tool 20        position relative to the anatomical structure A,    -   display the planned implant position and the surgical planning        in order to help the operator in choosing the best implant and        its position,    -   configuring a machining target position of a tool carrier.

The system for computer guided surgery of the present invention may beintegrated in the surgical device as control unit as described above orbe a processor configured to carry out the steps of the method of thepresent invention and to communicate to the surgical device by wireconnection or wirelessly.

The present invention further comprises a computer program product forcomputer guided surgery, the computer program product comprisinginstructions which, when the program is executed by a computer, causethe computer to carry out the steps of the method according to any oneof the embodiments described hereabove.

The computer program product to perform the method as described abovemay be written as computer programs, code segments, instructions or anycombination thereof, for individually or collectively instructing orconfiguring the processor or computer to operate as a machine orspecial-purpose computer to perform the operations performed by hardwarecomponents. In one example, the computer program product includesmachine code that is directly executed by a processor or a computer,such as machine code produced by a compiler. In another example, thecomputer program product includes higher-level code that is executed bya processor or a computer using an interpreter. Programmers of ordinaryskill in the art can readily write the instructions or software based onthe block diagrams and the flow charts illustrated in the drawings andthe corresponding descriptions in the specification, which disclosealgorithms for performing the operations of the method as describedabove.

The present invention further comprises a computer-readable storagemedium comprising instructions which, when the program is executed by acomputer, cause the computer to carry out the steps of the methodaccording to any one of the embodiments described hereabove.

According to one embodiment, the computer-readable storage medium is anon-transitory computer-readable storage medium.

Computer programs implementing the method of the present embodiments cancommonly be distributed to users on a distribution computer-readablestorage medium such as, but not limited to, an SD card, an externalstorage device, a microchip, a flash memory device, a portable harddrive and software websites. From the distribution medium, the computerprograms can be copied to a hard disk or a similar intermediate storagemedium. The computer programs can be run by loading the computerinstructions either from their distribution medium or their intermediatestorage medium into the execution memory of the computer, configuringthe computer to act in accordance with the method of this invention. Allthese operations are well-known to those skilled in the art of computersystems.

The instructions or software to control a processor or computer toimplement the hardware components and perform the methods as describedabove, and any associated data, data files, and data structures, arerecorded, stored, or fixed in or on one or more non-transitorycomputer-readable storage media. Examples of a non-transitorycomputer-readable storage medium include read-only memory (ROM),random-access memory (RAM), flash memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs,CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs,BD-Rs, BD-R LTHs, BD-REs, magnetic tapes, floppy disks, magneto-opticaldata storage devices, optical data storage devices, hard disks,solid-state disks, and any device known to one of ordinary skill in theart that is capable of storing the instructions or software and anyassociated data, data files, and data structures in a non-transitorymanner and providing the instructions or software and any associateddata, data files, and data structures to a processor or computer so thatthe processor or computer can execute the instructions. In one example,the instructions or software and any associated data, data files, anddata structures are distributed over network-coupled computer systems sothat the instructions and software and any associated data, data files,and data structures are stored, accessed, and executed in a distributedfashion by the processor or computer.

1-34. (canceled)
 35. A system for computer guided surgery comprisingperforming a transposition of an action planned in a virtual environmentwith respect to a virtual referential R_(P), to a physical actionperformed with a surgical tool in a real operating theatre environmentfor orthopedic surgery of a patient, said surgical tool is fixed to akinematic chain comprising a sensor unit having at least one sensorconfigured to follow in real time a spatial configuration of thekinematic chain; said system comprising: at least one input adapted toreceive at least one 3D image acquired from at least one 3D imagingsensor; said 3D image comprising at least one portion of a targetanatomical structure of the patient; at least one processor configuredto: calculate a transformation ^(C)T_(P) between the virtual referentialR_(p) and a target referential R_(C) and a transformation ^(C)T_(A)between an acquisition referential of the 3D imaging sensor R_(A) andthe target referential R_(C) by registration of a digital model of thetarget anatomical structure with the at least one portion of the targetanatomical structure comprised in the 3D image; apply the transformation^(C)T_(P) so as to register said digital model of the target anatomicalstructure in the target referential R_(C) so that each point comprisedin the digital model of the target anatomical structure has a knownposition in the target referential R_(C); calculate a transformation^(C)T_(O) between a referential of the surgical tool R_(O) and thetarget referential R_(C); apply the transformation ^(C)T_(O) to thereferential of the surgical tool R_(O) so as to know the position andspatial orientation of the surgical tool in the target referentialR_(C); so as to know the position and spatial orientation of saidsurgical tool in both the virtual referential R_(P) and the targetreferential R_(C) in order to reproduce the action planned in thevirtual referential R_(P) in the target referential R_(C).
 36. Thesystem according to claim 35, wherein the at least one processor isfurther configured to calculate the transformation ^(C)T_(A) by:defining a region of interest in the 3D image comprising said targetanatomical structure; registering said region of interest comprising thetarget anatomical structure to the digital model of the targetanatomical structure so as to determine ^(C)T_(A).
 37. The systemaccording to claim 35, wherein, when the kinematic chain comprises atleast one mechanical reference rigidly fixed to the target anatomicalstructure, and the at least one 3D image comprises at least one portionof the mechanical reference, the at least one processor is furtherconfigured to: receiving, from the sensor unit of the kinematic chain,data representative of the spatial configuration, in real time, of thekinematic chain; calculate a transformation ^(O)T_(M) between thereferential of the surgical tool R_(O) and the referential of themechanical reference R_(M) using the received data; calculate atransformation ^(M)T_(A) between the referential of the mechanicalreference R_(M) and the acquisition referential R_(A) by matching adigital model of the mechanical reference with the at least one portionof the mechanical reference comprised in the 3D image; so that the^(C)T_(O) transformation is obtained from the combination of thetransformations ^(O)T_(M), ^(M)T_(A) and ^(C)T_(A) between theacquisition referential R_(A), the mechanical reference referentialR_(M) and the target referential R_(C).
 38. The system according toclaim 37, wherein the at least one processor is further configured touse the sensor unit of the kinematic chain to track the movements of thetarget anatomical structure, rigidly fixed to the at least onemechanical reference, with respect to the surgical tool, so thatwhenever a deviation in the position and/or spatial orientation of thetarget anatomical structure is detected, the transposition of theplanned actions from the virtual environment to the real environment iscorrected for said deviation.
 39. The system according to claim 35,wherein the at least one 3D imaging sensor being fixed to the kinematicchain, the at least one processor is further configured to calculate atransformation ^(A)T_(O) between the referential of the surgical toolR_(O) and the acquisition referential of the 3D imaging sensor R_(A)from data obtained from the sensor unit of the kinematic chain so thatthe ^(C)T_(O) transformation is obtained from the combination of thetransformation ^(A)T_(O) and the transformation ^(C)T_(A) between theacquisition referential R_(A) and the target referential R_(C).
 40. Thesystem according to claim 35, wherein said kinematic chain consists of adeformable structure comprising multiple rigid elements connected byjoints and said kinematic chain further comprises sensors for measuringthe forces applied to its elements.
 41. The system according to claim35, wherein the acquisition of the 3D image received by the at least oneinput is carried out using at least two sensors and a projector to carryout an acquisition by stereovision or structured light.
 42. The systemaccording to claim 39, wherein the 3D imaging sensor fixed on thekinematic chain moves along a known trajectory and multiple 3D imagesare acquired along the trajectory, the at least one processor is furtherconfigured to jointly process multiple 3D images acquired along thetrajectory so as to use multiple 3D images for the registration with thedigital model of the target anatomical structure.
 43. The systemaccording to claim 39, wherein the at least one processor is furtherconfigured to track the movements of the target anatomical structurewith respect to the surgical tool using the 3D imaging sensor and avisual tracking algorithm, so that whenever a deviation in the positionand/or spatial orientation of the target anatomical structure isdetected, the transposition of the planned actions from the virtualenvironment to the real environment is corrected for said deviation. 44.The system according to claim 35, wherein the three-dimensional digitalmodel of the target anatomical structure is generated using 2D X-rayradiographies comprising the target anatomical structure, a statisticalshape model of the target anatomical structure and/or the 3D imageacquired intraoperatively by the 3D imaging sensor.
 45. The systemaccording to claim 35, wherein the three-dimensional digital model ofthe target anatomical structure is digitally modified to simulatemeasurement noise or the presence of cartilage, said modifications beingcalculated from training data or biomechanical simulation data.
 46. Thesystem according to claim35, wherein registration of the digital modelof the target anatomical structure with the at least one portion of thetarget anatomical structure comprised in the 3D image is a non-rigidtransformation.
 47. A computer-implemented method for guiding a surgicaltool in a real operating theatre environment suitable for orthopedicsurgery of a patient, said surgical tool is fixed to a kinematic chaincomprising a sensor unit having at least one sensor configured to followin real time a spatial configuration of the kinematic chain; said methodcomprising : receiving of at least one 3D image acquired from at leastone 3D imaging sensor; said 3D image comprising at least one portion ofa target anatomical structure of the patient; calculating atransformation ^(C)T_(P) between the virtual referential R_(P) and atarget referential R_(C) and a transformation ^(C)T_(A) between anacquisition referential of the 3D imaging sensor R_(A) and the targetreferential R_(C) by a registration of a digital model of the targetanatomical structure with the at least one portion of the targetanatomical structure comprised in the 3D image; applying thetransformation ^(C)T_(P) so as to register said digital model of thetarget anatomical structure in the target referential R_(C) so that eachpoint comprised in the digital model of the target anatomical structurehas a known position in the target referential R_(C); calculating atransformation ^(C)T_(O) between a referential of the surgical toolR_(O) and the target referential R_(C); applying the transformation^(C)T_(O) to the referential of the surgical tool R_(O) so as to knowthe position and spatial orientation of the surgical tool (20) in thetarget referential R_(C); so as to know the position and spatialorientation of said surgical tool (20) in both the virtual referentialR_(P) and the target referential R_(C) in order to reproduce the actionplanned in the virtual referential R_(P) in the target referentialR_(C).
 48. The method according to claim 47, wherein the calculation oftransformation ^(C)T_(A) comprises: defining a region of interest in the3D image comprising said target anatomical structure; registering saidregion of interest comprising the target anatomical structure to thedigital model of the target anatomical structure so as to determine^(C)T_(A).
 49. The method according to claim 47, wherein the kinematicchain comprises at least one mechanical reference rigidly fixed to thetarget anatomical structure and the at least one 3D image comprises atleast one portion of the mechanical reference; the method furthercomprises: receiving, from the sensor unit of the kinematic chain, datarepresentative of the spatial configuration, in real time, of thekinematic chain; calculating a transformation ^(O)T_(M) between thereferential of the surgical tool R_(O) and the referential of themechanical reference R_(M) using the received data; calculating atransformation ^(M)T_(A) between the referential of the mechanicalreference R_(M) and the acquisition referential R_(A) by matching adigital model of the mechanical reference with the at least one portionof the mechanical reference comprised in the 3D image; so that the ^(CT)_(O) transformation is obtained from the combination of thetransformations ^(O)T_(M), ^(M)T_(A) and ^(C)T_(A) between theacquisition referential R_(A), mechanical reference referential R_(M)and the target referential R_(C).
 50. The method according to claim 49,wherein the movements of the target anatomical structure with respect tosaid surgical tool are tracked by the sensor unit of the kinematic chainso that whenever a deviation in the position and/or spatial orientationof the target anatomical structure is detected, the transposition of theplanned actions from the virtual environment to the real environment iscorrected for said deviation.
 51. The method according to claim 47,wherein the at least one 3D imaging sensor is fixed to the kinematicchain the method further comprises calculating a transformation^(A)T_(O) between the referential of the surgical tool R_(O) and theacquisition referential of the 3D imaging sensor R_(A) from dataobtained from the sensor unit of the kinematic chain so that the^(C)T_(O) transformation is obtained from the combination of thetransformation ^(A)T_(O) and the transformation ^(C)T_(A) between theacquisition referential R_(A) and the target referential R_(C).
 52. Themethod according to claim 51, wherein the 3D imaging sensor fixed on thekinematic chain moves along a known trajectory and multiple 3D imagesare acquired along the trajectory, the method further comprises jointlyprocessing multiple 3D images acquired along the trajectory so as to usemultiple 3D images for the registration with the digital model of thetarget anatomical structure.
 53. The method according to claim 51,wherein the movements of the target anatomical structure with respect tosaid surgical tool are tracked by the 3D imaging sensor a visualtracking algorithm, so that whenever a deviation in the position and/orspatial orientation of the target anatomical structure is detected, thetransposition of the planned actions from the virtual environment to thereal environment is corrected for said deviation.
 54. Acomputer-readable storage medium comprising instructions which, whenexecuted by a computer, cause the computer to carry out the methodaccording to claim 47.